Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement performance benchmarks #446

Closed
wants to merge 14 commits into from
Closed

Implement performance benchmarks #446

wants to merge 14 commits into from

Conversation

testableapple
Copy link
Contributor

🔗 Issue Links

Resolves https://github.com/GetStream/ios-issues-tracking/issues/659

🎯 Goal

Implement Performance Benchmarks

🔖 Notion

📝 Summary

  • https://github.com/GetStream/stream-swift-performance-benchmarks stores the benchmarks
  • XCMetrics are executed on every PR (including release) by first PR's approval, manually on any branch and nightly by cron
  • XCMetrics won't be executed if there are no changes to the Sources folder
  • Collected metrics: :hitches_total_duration, :duration, :hitch_time_ratio, :frame_count, :frame_rate, :number_of_hitches
  • Metrics are collected from real devices on Firebase TestLab. Firebase offers us 5 runs per day (although, for some reason, sometimes it's even more than that), which should be enough most of the time (1 cron, 4 PRs). If this isn't enough, we can set up a paid account that offers free 30 minutes per day (1run ≈ 4min)
  • Our custom XCMetrics threshold equals to Xcode default one - 10%
  • Performance tests are executed on MessageList (with mocked static data: messages/images/videos/pdfs/giphies) and ChannelList (without mocks)
  • As far as I understood, Apple requires developers to run performance tests on the same machine + same iPhone (read more), which is impossible on CI. That's why there is no configured baseline in this PR, and the XCTests accordingly will always pass. Worries aside, because I implemented a custom calculation & assertion on the fastlane level.

🎨 Showcase

target metric branch release/4.48.1 performance status
MessageList Hitches total duration 5.0 ms 20 ms 75.0% 🔼 🟢
Duration 2.6 s 5 s 48.0% 🔼 🟢
Hitch time ratio 2.0 ms per s 10 ms per s 80.0% 🔼 🟢
Frame count 199 300 33.7% 🔼 🟢
Frame rate 78.4 fps 100 fps 21.6% 🔼 🟢
Number of hitches 0 1 100.0% 🔼 🟢
ChannelList Hitches total duration 10.0 ms 20 ms 50.0% 🔼 🟢
Duration 2.6 s 5 s 48.0% 🔼 🟢
Hitch time ratio 3.9 ms per s 10 ms per s 61.0% 🔼 🟢
Frame rate 74.7 fps 100 fps 25.3% 🔼 🟢
Number of hitches 0 1 100.0% 🔼 🟢

🧪 Manual Testing Notes

  • bundle exec fastlane xcmetrics

☑️ Contributor Checklist

  • I have signed the Stream CLA (required)
  • This change follows zero ⚠️ policy (required)
  • This change should be manually QAed
  • Changelog is updated with client-facing changes
  • New code is covered by unit tests
  • Comparison screenshots added for visual changes
  • Affected documentation updated (docusaurus, tutorial, CMS)

🎁 Meme

gif

@testableapple testableapple requested a review from a team as a code owner March 6, 2024 14:05
@testableapple testableapple added the 🤖 CI/CD Any work related to CI/CD label Mar 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 6, 2024 •
edited
Loading

6 Errors
🚫 Please use more than one word.
793e29e
🚫 Please use more than one word.
e54eb7d
🚫 Please use more than one word.
091328b
🚫 Please start subject with capital letter.
4ad1505
🚫 Please use more than one word.
4ad1505
🚫 Please use more than one word.
6a4bf8a
1 Warning
⚠️ Big PR

Generated by 🚫 Danger

@testableapple testableapple marked this pull request as draft March 6, 2024 17:09
@Stream-SDK-Bot
Copy link
Collaborator

StreamChat XCMetrics

target metric release/4.49.0 branch performance status
MessageList Hitches total duration 20 ms 30.9 ms -54.5% 🔽 🔴
Duration 5 s 2.5 s 50.0% 🔼 🟢
Hitch time ratio 10 ms per s 12.4 ms per s -24.0% 🔽 🔴
Frame rate 100 fps 79.3 fps 20.7% 🔼 🟢
Number of hitches 1 2 -100.0% 🔽 🔴
ChannelList Hitches total duration 20 ms 38.4 ms -92.0% 🔽 🔴
Duration 5 s 2.5 s 50.0% 🔼 🟢
Hitch time ratio 10 ms per s 15.4 ms per s -54.0% 🔽 🔴
Frame rate 100 fps 73.9 fps 26.1% 🔼 🟢
Number of hitches 1 3 -200.0% 🔽 🔴

@Stream-SDK-Bot
Copy link
Collaborator

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 30.0 ms -261.45% 🔽 🔴
Duration 2.6 s 2.48 s 4.62% 🔼 🟢
Hitch time ratio 3.3 ms per s 12.11 ms per s -266.97% 🔽 🔴
Frame rate 79 fps 79.29 fps -0.37% 🔽 🟡
Number of hitches 0.7 2.6 -271.43% 🔽 🔴
ChannelList Hitches total duration 10.8 ms 37.6 ms -248.15% 🔽 🔴
Duration 2.6 s 2.49 s 4.23% 🔼 🟢
Hitch time ratio 4.2 ms per s 15.03 ms per s -257.86% 🔽 🔴
Frame rate 75 fps 73.49 fps 2.01% 🔼 🟢
Number of hitches 0 3.6 -360.0% 🔽 🔴

@Stream-SDK-Bot
Copy link
Collaborator

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 24.2 ms -191.57% 🔽 🔴
Duration 2.6 s 2.5 s 3.85% 🔼 🟢
Hitch time ratio 3.3 ms per s 9.7 ms per s -193.94% 🔽 🔴
Frame rate 79 fps 79.33 fps -0.42% 🔽 🟡
Number of hitches 0.7 2.4 -242.86% 🔽 🔴
ChannelList Hitches total duration 10.8 ms 35.9 ms -232.41% 🔽 🔴
Duration 2.6 s 2.47 s 5.0% 🔼 🟢
Hitch time ratio 4.2 ms per s 14.54 ms per s -246.19% 🔽 🔴
Frame rate 75 fps 73.94 fps 1.41% 🔼 🟢
Number of hitches 0 3.2 -320.0% 🔽 🔴

@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Mar 8, 2024

Quality Gate Passed Quality Gate passed

Issues
47 New issues
0 Accepted issues

Measures
0 Security Hotspots
71.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@Stream-SDK-Bot
Copy link
Collaborator

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 49.2 ms -492.77% 🔽 🔴
Duration 2.6 s 2.49 s 4.23% 🔼 🟢
Hitch time ratio 3.3 ms per s 19.75 ms per s -498.48% 🔽 🔴
Frame rate 79 fps 78.57 fps 0.54% 🔼 🟢
Number of hitches 0.8 4.0 -400.0% 🔽 🔴
ChannelList Hitches total duration 10.8 ms 40.9 ms -278.7% 🔽 🔴
Duration 2.6 s 2.49 s 4.23% 🔼 🟢
Hitch time ratio 4.2 ms per s 16.41 ms per s -290.71% 🔽 🔴
Frame rate 76 fps 73.89 fps 2.78% 🔼 🟢
Number of hitches 0.8 3.8 -375.0% 🔽 🔴

@Stream-SDK-Bot
Copy link
Collaborator

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 60.1 ms -624.1% 🔽 🔴
Duration 2.6 s 2.49 s 4.23% 🔼 🟢
Hitch time ratio 3.3 ms per s 24.12 ms per s -630.91% 🔽 🔴
Frame rate 79 fps 77.79 fps 1.53% 🔼 🟢
Number of hitches 0.8 4.2 -425.0% 🔽 🔴
ChannelList Hitches total duration 10.8 ms 41.7 ms -286.11% 🔽 🔴
Duration 2.6 s 2.48 s 4.62% 🔼 🟢
Hitch time ratio 4.2 ms per s 16.82 ms per s -300.48% 🔽 🔴
Frame rate 76 fps 73.81 fps 2.88% 🔼 🟢
Number of hitches 0.8 3.6 -350.0% 🔽 🔴

@Stream-SDK-Bot
Copy link
Collaborator

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 32.5 ms -291.57% 🔽 🔴
Duration 2.6 s 2.49 s 4.23% 🔼 🟢
Hitch time ratio 3.3 ms per s 13.06 ms per s -295.76% 🔽 🔴
Frame rate 79 fps 78.88 fps 0.15% 🔼 🟢
Number of hitches 0.8 2.8 -250.0% 🔽 🔴
ChannelList Hitches total duration 10.8 ms 44.2 ms -309.26% 🔽 🔴
Duration 2.6 s 2.49 s 4.23% 🔼 🟢
Hitch time ratio 4.2 ms per s 17.79 ms per s -323.57% 🔽 🔴
Frame rate 76 fps 73.73 fps 2.99% 🔼 🟢
Number of hitches 0.8 4.0 -400.0% 🔽 🔴

@testableapple
Copy link
Contributor Author

Closing the PR for now. The branch is still active.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
🤖 CI/CD Any work related to CI/CD
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@testableapple @Stream-SDK-Bot @martinmitrevski